Papers with next-token prediction
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
C3D: Enhancing LLM Reasoning via Confidence-Guided Contrastive Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are prone to distraction by contextual information during reasoning tasks. |
| Approach: | They propose a decoding method that uses predicted logits to estimate the model's confidence. |
| Outcome: | The proposed method reveals how the model dynamically activates and adjusts its consideration of each premise as reasoning progresses. |
Beyond Tokens: Concept-Level Training Objectives for LLMs (2026.eacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained with a surprisingly narrow objective: predicting the next token in a sequence. |
| Approach: | They propose a shift from token-level to concept-level prediction where concepts group multiple surface forms of the same idea. |
| Outcome: | The proposed model improves on human-level models on diverse NLP benchmarks. |
Make Large Language Model a Better Ranker (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate robust capabilities across various fields . current list-wise approaches fail in ranking tasks due to misalignment between ranking objectives and next-token prediction . |
| Approach: | They propose a large language model framework with Aligned Listwise Ranking Objectives (ALRO) this framework provides explicit feedback in a listwise manner by introducing soft lambda loss . |
| Outcome: | The proposed model outperforms existing recommendation methods and embedding-based recommendations without additional computational burdens. |
MARIO: MAth Reasoning with code Interpreter Output - A Reproducible Pipeline (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack mathematical reasoning, a hurdle on the path to true artificial general intelligence. |
| Approach: | They propose a protocol for fine-tuning large language models with a Python code interpreter to enhance the text analysis of the LLMs. |
| Outcome: | The proposed protocol improves the performance of a 7B-parameter LLM on the GSM8K and MATH datasets while allowing for an outlier-free value model-based inference method. |
Frame-Semantic Knowledge Injection for Event-Level Inference in LLMs (2026.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) are fluent but often brittle when interpretation depends on external information. |
| Approach: | They propose a framework that injects frame-semantic knowledge into Large Language Models via LoRA. |
| Outcome: | The proposed framework can generalize beyond surface cues in large language models. |
When Will the Tokens End? Graph-Based Forecasting for LLMs Output Length (2025.acl-srw)
Copied to clipboard
Grzegorz Piotrowski, Mateusz Bystroński, Mikołaj Hołysz, Jakub Binkowski, Grzegorz Chodak, Tomasz Jan Kajdanowicz
| Challenge: | Large Language Models (LLMs) are typically trained to predict the next token in a sequence. However, their internal representations encode signals that go beyond immediate next-token prediction. |
| Approach: | They propose an aggregation-based model that combines hidden states from multiple transformer layers l 8, dots, 15 using element-wise operations such as mean or sum. |
| Outcome: | The proposed model reduces NMAE by over 50% on the Alpaca dataset. |
Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM’s Nest (2025.acl-long)
Copied to clipboard
| Challenge: | Massive high-quality data, both pre-training raw texts and post-training annotations, have been carefully prepared to incubate advanced large language models (LLMs). |
| Approach: | They propose to reframe next-token prediction into extraction for tokens already present in the context of LLMs by reframing next-tongue prediction into IE models. |
| Outcome: | The proposed model learns 102.6M extractive data converted from pre-training and post-training data with better performance than existing pre-trained IE models. |
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe (2026.acl-long)
Copied to clipboard
| Challenge: | Efforts to improve instruction tuning often focus on higher-quality supervised fine-tuning datasets, typically requiring data filtering with proprietary LLMs or human annotation. |
| Approach: | They propose a Mixup-based recipe that elevates LLM instruction tuning without relying on well-curated datasets. |
| Outcome: | The proposed model improves instruction-following and healthcare-specific tasks with consistent improvements across LLM families and SFT datasets. |
CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual Large Language Models (MLLMs) exhibit strong generalization across languages, yet they remain prone to hallucinations due to training data imbalances. |
| Approach: | They propose a cross-lingual Chain-of-Thought framework that enhances cross-linguistic alignment . the framework guides the model to reason in a high-resource language before generating answers in low-resourced language. |
| Outcome: | The proposed framework reduces hallucination rates by up to 62% and significantly improves factual knowledge transfer across language pairs. |
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used in real-world industrial scenarios for various natural language processing tasks, but their high inference cost makes their deployment impractical, necessitating the use of smaller models. |
| Approach: | They propose a continual pre-training technique that generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. |
| Outcome: | The proposed technique improves small LLMs’ domain adaptability for business conversational tasks, compared with traditional methods that rely on next-token prediction. |
Differentially Private Next-Token Prediction of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming increasingly important for ensuring privacy, but DP-SGD overestimates an adversary’s capabilities in having white box access. |
| Approach: | They propose a private prediction protocol for next-token prediction that utilizes the inherent stochasticity of next-toning and a public model to achieve Differential Privacy. |
| Outcome: | The proposed protocol outperforms DP-SGD and DP training methods for privacy on large datasets. |
POP: Prefill-Only Pruning for Efficient Large Model Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing structured pruning methods suffer from significant accuracy degradation . Existing pruning methods are expensive and require specialized hardware and kernels to perform . |
| Approach: | They propose a stage-agnostic pruning approach that overlooks asymmetric roles between prefill and decode stages. |
| Outcome: | The proposed pruning approach achieves 1.37 speedup in prefill latency with minimal performance loss. |
Reinforcement Learning on Pre-Training Data (2026.acl-long)
Copied to clipboard
Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Kun Li, Haoyuan Wu, null Wujiajia, Zihao Zheng, Chenchen Zhang, Kun Shi, Xue Gong, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang xu, Zheng Fang, Shuai LI, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bochao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Bo Zhou, Di Wang
| Challenge: | Recent progress in large language models is driven by scaling of training compute through pre-training with nexttoken prediction (NTP) or post-training (RL) Pre-training using NTP enables models to acquire extensive knowledge and skills from general data, but it suffers from data inefficiency and catastrophic forgetting in continual learning settings. |
| Approach: | They propose to scale training compute through pre-training with next-token prediction (NTP) or post-training by scaling reinforcement learning (RL) to improve learning from general data. |
| Outcome: | Experiments on multiple benchmarks and models show that the proposed approach improves continual pre-training and provides a strong foundation for post-training on Qwen3-8B-Base. |
Coherence boosting: When your pretrained language model is not paying enough attention (2022.acl-long)
Copied to clipboard
| Challenge: | Long-range semantic coherence remains a challenge in automatic language generation and understanding. |
| Approach: | They propose a procedure that increases a model’s focus on a long context by distributional analyses of generated ordinary text and dialog responses. |
| Outcome: | The proposed procedure increases the model's focus on a long context. |
Flashback: Memory Mechanism for Enhancing Memory Efficiency and Speed in Deep Sequential Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing deep sequential processing models have problems with memory degradation and inaccurate gradient backpropagation. |
| Approach: | They propose a Flashback property that preserves memory as an identity mapping until it is overwritten by a hidden state at a different time step. |
| Outcome: | The proposed model can be implemented in Transformers and Mamba, and it performs well. |
Language Models Struggle to Use Representations Learned In-Context (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that large language models are capable of inducing rich representations of data that are seen in-context . a novel task, adaptive world modeling, shows that even the most performant LLMs cannot reliably leverage novel semantics defined in-constitut. |
| Approach: | They propose to use in-context representations to induce rich representations of data . they also propose to probe models using a novel task to enable flexible deployment . |
| Outcome: | The proposed model can use in-context representations to complete simple downstream tasks. |
An Analysis and Mitigation of the Reversal Curse (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research observes a phenomenon in large language models called the "reversal curse" when dealing with two entities, LLMs excel in handling sequences in the form of "aRb" but when asked "who is Mary Lee Pfeiffer's son?" the LLM exhibits considerable confusion and fails to provide a as the answer . |
| Approach: | They conduct the first-ever study of how the reversal curse happens in large language models . they find that LLMs excel in handling sequences in the form of "aRb" but struggle to provide a satisfactory answer when asked "who is Mary Lee Pfeiffer's son?" |
| Outcome: | The proposed study shows that the reversal curse can stem from specific training objectives . the study also shows that a reverse query can be difficult to understand . |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |
DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can replicate insecure patterns from training data. |
| Approach: | They propose a framework that leverages distributed security-relevant cues by aggregating representations from multiple upper layers via an attention-based module. |
| Outcome: | Experiments show that the framework improves the secure-and-correct generation rate by 11.9% over baselines. |
Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using a method to identify next-token neurons, we find that some attention heads recognize contexts relevant to predicting a token and activate a downstream token-predicting neuron accordingly. |
| Approach: | They propose a method to identify next-token neurons and determine the upstream attention heads responsible for their activity in LLMs. |
| Outcome: | The proposed method identifies next-token neurons, finds prompts that highly activate them, and determines the upstream attention heads responsible. |
VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show VocalNet outperforms existing open-source speech LLMs despite limited training data. |
| Approach: | They propose a scalable and model-agnostic training framework and a novel multi-token prediction paradigm for speech generation. |
| Outcome: | The proposed model outperforms open-source speech LLMs while outperforming existing open-sourced models. |
Visual Self-Refinement for Autoregressive Models (2025.findings-emnlp)
Copied to clipboard
Jiamian Wang, Ziqi Zhou, Chaithanya Kumar Mummadi, Sohail Dianat, Majid Rabbani, Raghuveer Rao, Chen Qiu, Zhiqiang Tao
| Challenge: | Autoregressive models excel in sequential modeling but the spatial nature of visual signals conflicts with the sequential dependencies of next-token prediction, leading to suboptimal results. |
| Approach: | They propose a plug-and-play refinement module to enhance the spatial correspondence modeling within the generated visual sequence. |
| Outcome: | The proposed module enhances vision-language modeling under a shared sequential prediction framework. |
What’s in a prompt? Language models encode literary style in prompt embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode textual information using high-dimensional latent spaces . many studies have investigated how conceptual content of words translates into geometrical relationships between their vector representations . |
| Approach: | They use literary pieces to show that intangible, rather than factual, aspects of the prompt are contained in deep representations. |
| Outcome: | The results show that word-to-vec(tor) embeddings are more complex than other models. |
Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | federated learning (FL) fine-tunes large language models with local data, but organizations are reluctant to share local data. |
| Approach: | They propose a framework for fine-tuning large language models with local data . they propose centralized fine- tuning with local datasets is a good idea . |
| Outcome: | The proposed framework allows clients to retain local data while sharing only model parameters for training. |
Predicting Through Generation: Why Generation Is Better for Prediction (2025.acl-long)
Copied to clipboard
Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat, Chun-Nam Yu, Mojtaba Soltanalian, Ivan Garibay, Ozlem Garibay, Chen Chen, Niloofar Yousefi
| Challenge: | Large Language Models (LLMs) are increasingly used for predictive tasks such as classification and regression. |
| Approach: | They propose a framework that generates output tokens from mas-sive text corpora and a task adapter to ensure consistency between token generation and final prediction. |
| Outcome: | The proposed framework outperforms baseline models on classification and regression benchmarks and the proposed framework consistently outperformed standard baseline models. |
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only LMs fail to solve the path-star task above 1/D chance due to a learned shortcut that absorbs training supervision. |
| Approach: | They propose a path-star task which is a minimal example of searching over a graph with D arms rooted at a single start node and a query to generate the arm with t from s to t. |
| Outcome: | The proposed task is solvable via decoder-only LMs and its minimal nature prevents its decomposition. |
All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) perform well on a multitude of computational tasks, yet their inner workings remain unclear. |
| Approach: | They propose two techniques to inhibit input-specific token computations in initial layers . they propose a transformer that allows for any token to immediately access all preceding tokens . |
| Outcome: | The proposed algorithms can perform on a variety of mental math tasks with high accuracy and transfer across models. |
Rejection-to-Acceptance Transition: Model Editing-Based Jailbreak Backdoor Injection Not Limited to Few Output Tokens (2026.findings-acl)
Copied to clipboard
Shiji Yang, Min Cai, Hao Xiong, Congyao Mei, Haodong Zou, Shicheng Tan, Jie Chen, Fulan Qian, Shu Zhao
| Challenge: | Existing methods for jailbreaking LLMs are implemented by binding backdoors to predefined phrases as first few output tokens, inducing the LLM’s next-token prediction to produce continuous responses. |
| Approach: | They propose a model editing-based jailbreak backdoor attack that hijacks LLM representations into a acceptance domain rather than binding to a few output tokens. |
| Outcome: | The proposed model editing method outperforms existing methods, showing stronger jailbreak capabilities across LLMs and datasets. |
Planning-Aware Code Infilling via Horizon-Length Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to fill-in-the-middle (FIM) often fail to generate content that aligns well with the surrounding context. |
| Approach: | They propose a training objective that teaches models to predict the number of remaining middle tokens at each step. |
| Outcome: | The proposed training objective improves FIM performance by up to 24% on diverse benchmarks across file-level and repository-level. |